Optimization Strategies and Backpropagation Neural Networks

Adaptive learning rate algorithms try to decrease the error at each iteration by searching a local minimum with small weight steps, which are usually constrained by highly problemdependent heuristic learning parameters. Based on the idea of the decrease of the error function at each iteration we suggest monotone learning strategies that guarantee convergence to a minimizer of the error function without using highly problem-dependent heuristics. Furthermore, we introduce the idea of nonmonotone learning that provides fast, stable and reliable training and we test both approaches on simulation experiments.